1. Field of the Invention
The present invention relates to an image processing method, image processing apparatus, computer program, and computer readable storage medium and, more particularly, to a technique for processing a plurality of pieces of image data captured at different image magnifications.
2. Description of the Related Art
A variety of surveillance systems based on an image provided by an image pickup device have been proposed, such as surveillance systems using a wide-angle lens such as a fisheye lens.
The advantages of the surveillance system using the wide-angle lens are that a single image pickup device monitors a wide area in a single captured image and that the cost required for the investment of the image pickup device per surveillance area is low. The surveillance system reliably captures an image of any trespasser in a wide surveillance area and helps a user to easily recognize the position of the trespasser.
FIG. 1 illustrates an office in which a single wide-angle camera 1301 mounted on a high place in the office monitors a relatively wide surveillance area 1302.
A trespasser 1303 is now entering the office. Because of the wide surveillance area of the wide-angle camera 1301, the system can view many office instruments such as desks and doors in addition to the incoming trespasser 1303 in the same image. A user immediately visibly recognizes the position of the trespasser 1303. The system keeps the moving trespasser 1303 within the wide field of view thereof. It is difficult for the trespasser 1303 to disappear from the field of view.
A surveillance system employing a wide-angle lens with a 360 degree full circle surveillance capability has been proposed. Such a system incorporates an optical mirror element in an image pickup system and employs a technique that corrects, through image processing, distortion of a captured image that becomes larger as the image pickup optical system has a wider field of view.
Although the wide-angle lens surveillance system has the above-mentioned advantages, the image of the trespasser 1303 in the wide surveillance area becomes smaller and more difficult to identify.
FIG. 2 shows one scene of the video taken by the wide-angle camera 1301 where the trespasser 1303 enters the office. As shown in FIG. 2, the wider the surveillance area, the more easily the user recognizes the position of the trespasser 1303. However, it is more difficult for the user to recognize the details, such as the face, expression, and appearance of the trespasser 1303.
FIG. 3 illustrates another such a surveillance system. The surveillance area 1302 monitored by the wide-angle camera 1301 illustrated in FIG. 1 is segmented into three surveillance areas. The three surveillance areas are respectively monitored by three cameras 1501, each having a standard lens.
The three cameras 1501 independently monitor the respective surveillance areas (a surveillance area 1502, for example), and captured images are then presented on respective monitors on a one camera to one video screen basis. Some controllers receive videos from a plurality of cameras in a multi-channel input port, and then present the videos on a multi-window screen on a single monitor.
When a trespasser 1303 comes into the office, one of the three cameras (a center camera) shown in FIG. 3 captures the image of the trespasser 1303, and a monitor linked to the center camera presents a screen shown in FIG. 4. The screen shown in FIG. 4 may be the one presented on one channel of multi-channel video from the three cameras.
A comparison of FIG. 4 with FIG. 2 reveals that the surveillance system shown in FIG. 4 monitors the image of the trespasser 1303 in a larger size, and outperforms the wide-angle lens surveillance system shown in FIG. 2 in the recognition of details, such as the face, expression, and appearance of the trespasser 1303.
A user (such as a security guard) of the surveillance system must view a plurality of monitors or a multi-window screen at a time.
Japanese Patent Laid-Open No. 2000-295600 discloses a technique to overcome this problem. According to the disclosure, a plurality of cameras are used to monitor a surveillance area. If a moving object is detected from a video, video containing the moving object is presented on a single monitor.
FIG. 5 illustrates another surveillance system, disclosed in Japanese Patent Laid-Open No. 2000-32319. A camera 1701, which may be panned, tilted, and zoomed to change the field of view (here simply referred to as a zoom camera), is installed in addition to the wide-angle camera 1301 in the office illustrated in FIG. 1. The wide-angle camera 1301 monitors a wide area while the zoom camera 1701 takes an expanded image that presents more clearly the face and appearance of the trespasser 1303.
FIGS. 6A and 6B illustrate an image captured by the wide-angle camera 1301, and the zoom camera 1701, respectively, shown in FIG. 5.
FIG. 6A illustrates an image captured by the wide-angle camera 1301 and is identical to the image illustrated in FIG. 2. FIG. 6B illustrates an image captured by the zoom camera 1701. As with the surveillance system having a plurality of image pickup devices, this surveillance system may present images on monitors respectively linked with cameras, or a multi-window screen on a single monitor using a multi-channel controller.
This system advantageously satisfies one requirement that the position of the trespasser is easily recognized in a wide-angle surveillance image provided by the wide-angle lens camera, and another requirement that the face, expression, and appearance of the trespasser are clearly viewed in an expanded image of the zoom camera.
Japanese Patent Laid-Open No. 2000-32319 discloses a system in which images captured by a wide-angle camera and a zoom camera are transferred and displayed on an apparatus connected to a network. Also proposed in the same disclosure is a surveillance system in which a user watches an image on the wide-angle camera while watching an expanded image of a desired object by operating the apparatus to pan, tilt, and zoom the zoom camera at the same time.
Rather than manually operating a zoom camera, Japanese Patent Laid-Open No. 2000-295200 discloses a surveillance system which automatically detects a moving object from a wide-angle image, and automatically pans, tilts, or zooms the zoom camera in response to a position and a size of the moving object to obtain an expanded image of the moving object.
The foregoing conventional methods have the following problems.
In the surveillance system using the wide-angle lens such as a fisheye lens, the size of the image of the trespasser is small with respect to the surveillance area image and the user has difficulty in recognizing the details of the trespasser, such as the face. When the trespasser is detected, the area of the trespasser is expanded by performing an expansion and interpolation process on a digital image. However, since the amount of information (a frequency component in the image) present in the image is determined during sampling, the amount of information is not increased subsequent to the expansion process.
In other words, the expansion process does not show any detail if it has been invisible in the image prior to the expansion process. Since an apparent frequency component of the image shifts to a lower frequency with the expansion rate, an expanded image becomes simply blurred. If no interpolation process is performed, the image typically becomes a block-mosaic image.
Japanese Patent Laid-Open No. 2000-295600 discloses an improved manner of presenting the image to ease the fatigue of the security guard who must continuously view the image. As the surveillance area becomes wide, more cameras are required, and costs increase.
The security guard or other user may have difficulty in discerning the position of a trespasser because each camera has a small surveillance area. As the trespasser moves, the system switches from one camera to another to keep the trespasser to within the field of view with each camera covering a different surveillance area. In this arrangement, the user may have difficulty in identifying the trespasser. The disclosure mentions a method of obtaining an expanded image by image processing a pickup image from the camera. For this reason, a great deal of effort is required to construct an appropriate environment including the maximum effective expansion rate effective for recognizing the object image, focal length of the camera (or the field of view), and distance between installed cameras, depending on monitoring positions and recognition precision in identifying the trespasser.
Although the surveillance system employing a plurality of cameras having different image magnifications optically overcomes the drawbacks of the conventional art, two video signals, one from the wide-angle camera and the other from the zoom camera, must be transmitted to the network, requiring a large amount of bandwidth.
As in FIG. 2 and FIG. 6A, most of the video from the wide angle camera, except of course for the trespasser, does not change, even during the trespassing, and thus many image frames are transmitted redundantly. There is room for improvement in terms of effective and efficient video transmission.
From this standpoint, Japanese Patent Laid-Open No. 2000-32319 sets a frame rate for transmitting a zoom camera expanded video to be higher than a frame rate for transmitting a wide-angle camera video so that the zoom camera expanded image moves more smoothly. This is because the zoom camera expanded image is considered more important than the wide-angle camera image. However, the reduction of the frame rate of the wide-angle camera video negatively affects some frames imaging the action of the trespasser, leading to a problem in as a surveillance system.
In particular, a criminal act is typically performed very quickly. The reduction of the frame rate may eliminate frames, which could otherwise become an important evidence later.
In the surveillance system having a plurality of cameras having different image magnifications, the background images must be prepared beforehand to extract an object image. As discussed in Japanese Patent Laid-Open No. 2000-295600 to effectively extract an object image, motion must be detected. It is difficult to efficiently extract a target object image from multiple streams of video, since a captured background image may have an image magnification different from a desired image magnification. Therefore, extracting a desired image of an object at a desired magnification can be difficult to obtain.